{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment Analyzer For Movie Reviews\n",
    "\n",
    "The scraping process took 2 hours to finish. In the end, I was able to obtain all needed 28 variables for 5043 movies and 4906 posters (998MB), spanning across 100 years in 66 countries. There are 2399 unique director names, and thousands of actors/actresses. Below are the 28 variables:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from __future__ import division, print_function, absolute_import\n",
    "\n",
    "import tflearn\n",
    "import tensorflow as tf\n",
    "from tflearn.data_utils import to_categorical, pad_sequences\n",
    "from tflearn.datasets import imdb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function input_data in module tflearn.layers.core:\n",
      "\n",
      "input_data(shape=None, placeholder=None, dtype=tf.float32, data_preprocessing=None, data_augmentation=None, name='InputData')\n",
      "    Input Data.\n",
      "    \n",
      "    `input_data` is used as a data entry (placeholder) of a network.\n",
      "    This placeholder will be feeded with data when training\n",
      "    \n",
      "    This layer is used to keep track of the network inputs, by adding the\n",
      "    placeholder to INPUTS graphkey. TFLearn training functions may retrieve\n",
      "    those variables to setup the network training process.\n",
      "    \n",
      "    Input:\n",
      "        List of `int` (Shape), to create a new placeholder.\n",
      "            Or\n",
      "        `Tensor` (Placeholder), to use an existing placeholder.\n",
      "    \n",
      "    Output:\n",
      "        Placeholder Tensor with given shape.\n",
      "    \n",
      "    Arguments:\n",
      "        shape: list of `int`. An array or tuple representing input data shape.\n",
      "            It is required if no placeholder provided. First element should\n",
      "            be 'None' (representing batch size), if not provided, it will be\n",
      "            added automatically.\n",
      "        placeholder: A Placeholder to use for feeding this layer (optional).\n",
      "            If not specified, a placeholder will be automatically created.\n",
      "            You can retrieve that placeholder through graph key: 'INPUTS',\n",
      "            or the 'placeholder' attribute of this function's returned tensor.\n",
      "        dtype: `tf.type`, Placeholder data type (optional). Default: float32.\n",
      "        data_preprocessing: A `DataPreprocessing` subclass object to manage\n",
      "            real-time data pre-processing when training and predicting (such\n",
      "            as zero center data, std normalization...).\n",
      "        data_augmentation: `DataAugmentation`. A `DataAugmentation` subclass\n",
      "            object to manage real-time data augmentation while training (\n",
      "            such as random image crop, random image flip, random sequence\n",
      "            reverse...).\n",
      "        name: `str`. A name for this layer (optional).\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(tflearn.input_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#IMDB Dataset loading\n",
    "#load_data in module tflearn.datasets.imdb:\n",
    "#extension is picke which means a bit steam, so it make it easier to convert to other \n",
    "#python objects like list, tuple later \n",
    "#param n_words: The number of word to keep in the vocabulary.All extra words are set to unknow (1).\n",
    "#param valid_portion: The proportion of the full train set used for the validation set.\n",
    "\n",
    "train, test, _ = imdb.load_data(path='imdb.pkl', n_words=10000, valid_portion=0.1)\n",
    "trainX, trainY = train\n",
    "testX, testY = test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Data preprocessing\n",
    "# Sequence padding\n",
    "#function pad_sequences in module tflearn.data_utils:\n",
    "# padding is necessary to have consistency in our input dimensionality\n",
    "trainX = pad_sequences(trainX, maxlen=100, value=0.)\n",
    "testX = pad_sequences(testX, maxlen=100, value=0.)\n",
    "\n",
    "#Now we can convert output to binaries where 0 means Negative and 1 means Positive\n",
    "# Converting labels to binary vectors\n",
    "trainY = to_categorical(trainY, nb_classes=2)\n",
    "testY = to_categorical(testY, nb_classes=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[   0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0   16  586   32  885   17   39   68   31 2994 2389\n",
      "   328    4]\n",
      " [   0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    1    2    1  139    6  130    1    5    6   25  105\n",
      "  4730   40]\n",
      " [   0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0   30  287  142 2216  707 3763   20   68   57   30   37  309\n",
      "    14    4]\n",
      " [   0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0 6691    1   10  333   10   17   27    4   34  181    6 1418\n",
      "   256    4]\n",
      " [   0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0  224    3\n",
      "   371    3    1    4  128   37   16   90   48 1247    8   79  294  913\n",
      "  1709    4]\n",
      " [   0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0   17   10    2 6409\n",
      "    25  325    7 3161    4 2588 1977  176    3   26   50   35   70 1050\n",
      "   395    4]\n",
      " [   0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0   70  922   87  394   25\n",
      "     3  129  883   53  457   86  879   87   70  297   42   41   86 1752\n",
      "    14   40]\n",
      " [   0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0   16  104   32  138  152\n",
      "    16   49   17   25   48   89    3   26   16  128   91 1445    7  164\n",
      "    14    4]\n",
      " [   0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    2  118   35 6186    5    2\n",
      "   242   10  397    4   14   20    6  456    7    2 1938    7    1    5\n",
      "     1    4]]\n",
      "(100,)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "numpy.int32"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(trainX[1:10,:])\n",
    "print(trainX[0,:].shape)\n",
    "type(trainX[0,0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "Input 'split_dim' of 'Split' Op has type float32 that does not match expected type of int32.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m/home/amanullahtariq/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py\u001b[0m in \u001b[0;36mapply_op\u001b[0;34m(self, op_type_name, name, **keywords)\u001b[0m\n\u001b[1;32m    490\u001b[0m                 \u001b[0mas_ref\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minput_arg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_ref\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 491\u001b[0;31m                 preferred_dtype=default_dtype)\n\u001b[0m\u001b[1;32m    492\u001b[0m           \u001b[0;32mexcept\u001b[0m \u001b[0mTypeError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0merr\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/amanullahtariq/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py\u001b[0m in \u001b[0;36minternal_convert_to_tensor\u001b[0;34m(value, dtype, name, as_ref, preferred_dtype)\u001b[0m\n\u001b[1;32m    715\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mret\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 716\u001b[0;31m           \u001b[0mret\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconversion_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mas_ref\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mas_ref\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    717\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/amanullahtariq/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py\u001b[0m in \u001b[0;36m_TensorTensorConversionFunction\u001b[0;34m(t, dtype, name, as_ref)\u001b[0m\n\u001b[1;32m    588\u001b[0m         \u001b[0;34m\"Tensor conversion requested dtype %s for Tensor with dtype %s: %r\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 589\u001b[0;31m         % (dtype.name, t.dtype.name, str(t)))\n\u001b[0m\u001b[1;32m    590\u001b[0m   \u001b[0;32mreturn\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: Tensor conversion requested dtype int32 for Tensor with dtype float32: 'Tensor(\"Embedding/Reshape:0\", shape=(?, 128), dtype=float32)'",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-6-c14520cc65d4>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     14\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m \u001b[0;31m# LAYER-2\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 16\u001b[0;31m \u001b[0mnet\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtflearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0membedding\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnet\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput_dim\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m10000\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moutput_dim\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m128\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     17\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     18\u001b[0m \u001b[0;31m# LSTM = long short term memory\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/amanullahtariq/anaconda3/lib/python3.5/site-packages/tflearn/layers/embedding_ops.py\u001b[0m in \u001b[0;36membedding\u001b[0;34m(incoming, input_dim, output_dim, weights_init, trainable, restore, name)\u001b[0m\n\u001b[1;32m     54\u001b[0m         \u001b[0minference\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtranspose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minference\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     55\u001b[0m         \u001b[0minference\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreshape\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minference\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mshape\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moutput_dim\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 56\u001b[0;31m         \u001b[0minference\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_inputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minference\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     57\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     58\u001b[0m     \u001b[0;31m# TODO: easy access those var\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/amanullahtariq/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py\u001b[0m in \u001b[0;36msplit\u001b[0;34m(value, num_or_size_splits, axis, num, name)\u001b[0m\n\u001b[1;32m   1214\u001b[0m   \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnum_or_size_splits\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msix\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minteger_types\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1215\u001b[0m     return gen_array_ops._split(\n\u001b[0;32m-> 1216\u001b[0;31m         split_dim=axis, num_split=num_or_size_splits, value=value, name=name)\n\u001b[0m\u001b[1;32m   1217\u001b[0m   \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1218\u001b[0m     \u001b[0msize_splits\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconvert_to_tensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnum_or_size_splits\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/amanullahtariq/anaconda3/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py\u001b[0m in \u001b[0;36m_split\u001b[0;34m(split_dim, value, num_split, name)\u001b[0m\n\u001b[1;32m   3424\u001b[0m   \"\"\"\n\u001b[1;32m   3425\u001b[0m   result = _op_def_lib.apply_op(\"Split\", split_dim=split_dim, value=value,\n\u001b[0;32m-> 3426\u001b[0;31m                                 num_split=num_split, name=name)\n\u001b[0m\u001b[1;32m   3427\u001b[0m   \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3428\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/amanullahtariq/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py\u001b[0m in \u001b[0;36mapply_op\u001b[0;34m(self, op_type_name, name, **keywords)\u001b[0m\n\u001b[1;32m    507\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0minput_arg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtype\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mtypes_pb2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDT_INVALID\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    508\u001b[0m               raise TypeError(\"%s expected type of %s.\" %\n\u001b[0;32m--> 509\u001b[0;31m                               (prefix, dtypes.as_dtype(input_arg.type).name))\n\u001b[0m\u001b[1;32m    510\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    511\u001b[0m               \u001b[0;31m# Update the maps with the default, if needed.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: Input 'split_dim' of 'Split' Op has type float32 that does not match expected type of int32."
     ]
    }
   ],
   "source": [
    "# Network building\n",
    "#function input_data in module tflearn.layers.core:\n",
    "#`input_data` is used as a data entry (placeholder) of a network.\n",
    "#This placeholder will be feeded with data when training\n",
    "# shape: list of `int`. An array or tuple representing input data shape.\n",
    "#            It is required if no placeholder provided. First element should\n",
    "#            be 'None' (representing batch size), if not provided, it will be\n",
    "#            added automatically.\n",
    "net = tflearn.input_data([None, 100])\n",
    "\n",
    "# LAYER-1\n",
    "# input_dim = 10000, since that how many words we loaded from our dataset\n",
    "# output_dim = 128, number of outputs of resulting embedding\n",
    "\n",
    "# LAYER-2\n",
    "net = tflearn.embedding(net, input_dim=10000, output_dim=128)\n",
    "\n",
    "# LSTM = long short term memory\n",
    "# this layer allow us to remember our data from the beginning of the sequences which \n",
    "# allow us to improve our prediction\n",
    "# dropout = 0.8, is the technique to prevent overfitting, which randomly switch on and off different \n",
    "# pathways in our network\n",
    "net = tflearn.lstm(net, 128, dropout=0.8)\n",
    "\n",
    "# LAYER-3: Our next layer is fully connected, this means every neuron in the previous layer is connected to \n",
    "# the every neuron in this layer\n",
    "# Good way of learning non-linear combination\n",
    "# activation: take in vector of input and squash it into vector of output probability between \n",
    "# 0 and 1.\n",
    "net = tflearn.fully_connected(net, 2, activation='softmax')\n",
    "\n",
    "# LAYER-4: \n",
    "# adam : perform gradient descent\n",
    "# categorical_crossentropy: find error between predicted output and original output \n",
    "net = tflearn.regression(net, optimizer='adam', learning_rate=0.0001, loss='categorical_crossentropy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Training\n",
    "# Initialize with tflearn Deep Neural Network Model.\n",
    "# tensorboard_verbose: `int`. Summary verbose level, it accepts\n",
    "#          different levels of tensorboard logs:\n",
    "#          0: Loss, Accuracy (Best Speed).\n",
    "#          1: Loss, Accuracy, Gradients.\n",
    "#          2: Loss, Accuracy, Gradients, Weights.\n",
    "#          3: Loss, Accuracy, Gradients, Weights, Activations, Sparsity.\n",
    "#              (Best visualization)\n",
    "model = tflearn.DNN(net, tensorboard_verbose=0)\n",
    "model.fit(trainX, trainY, validation_set=(testX, testY), show_metric=True,batch_size=32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "help(tflearn.DNN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "testY\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "predictionsY = model.predict(testX)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.scatter(testX[:,1], testX[:,2])\n",
    "plt.xlabel('X')\n",
    "plt.ylabel('Y')\n",
    "plt.title('Imdb Film Dataset')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Citation\n",
    "\n",
    "@InProceedings{maas-EtAl:2011:ACL-HLT2011,\n",
    "  author    = {Maas, Andrew L.  and  Daly, Raymond E.  and  Pham, Peter T.  and  Huang, Dan  and  Ng, Andrew Y.  and  Potts, Christopher},\n",
    "  title     = {Learning Word Vectors for Sentiment Analysis},\n",
    "  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},\n",
    "  month     = {June},\n",
    "  year      = {2011},\n",
    "  address   = {Portland, Oregon, USA},\n",
    "  publisher = {Association for Computational Linguistics},\n",
    "  pages     = {142--150},\n",
    "  url       = {http://www.aclweb.org/anthology/P11-1015}\n",
    "}\n",
    "\n",
    "# References\n",
    "\n",
    "Potts, Christopher. 2011. On the negativity of negation. In Nan Li and\n",
    "David Lutz, eds., Proceedings of Semantics and Linguistic Theory 20,\n",
    "636-659."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
